In practice, however, REST does not fit every problem. gRPC is a more general protocol that can be used to call arbitrary functions across process or network boundaries. In this article I describe how to use the Go programming language to develop efficient gRPC services that are easily accessible from other platforms such as .NET and C#.
STAY TUNED!
Learn more about API Conference
Why gRPC and not REST?
Many microservices don’t care about storing data in databases, but offer different functions. Accordingly, it is difficult to force them into the corset of REST. Is the activation of a heating function on an IoT device an insert or update? How does a service for restarting a VM fit into the concept of REST? How do you model a service for sending SMS messages so that the API can be easily understood by others? The concept of RPC is ancient and more general than REST. All operations are traced back to a collection of commands that receive parameters and return the result.
The advantage of this generic approach is that it can be applied to a larger number of problems without having to squeeze them into a poorly fitting architectural principle. The disadvantage is that each RPC interface can be modeled differently, while RESTful APIs – where they conceptually fit – are more consistent and therefore easier to understand and use. In practice, I recommend avoiding discussions about one or the other. There is nothing to be said against offering a cloud service with a RESTful API and a gRPC API in case of doubt. The logic in the background should be the same, only the communication protocol can be chosen by the user depending on the technology and use case.
Of course, a modern RPC system like gRPC is very different from its ancestors from the 70s. It has become much more powerful and supports, for example, the efficient transmission of complex data structures using protocol buffers (ProtoBuf) and streaming in both communication directions. Frameworks for developing gRPC servers and generating gRPC clients are available for all widely used programming languages. In my experience the tools for code generation are better and more stable than many of the code generators for Swagger aka Open API. In addition, gRPC is based on HTTP/2, which gives a comparatively better performance for communication between microservices.
After this short introduction, I will not go into the basics of gRPC and ProtoBuf in this article, as they have already been described elsewhere. Let’s now take a look at the development of gRPC services with Go.
Why Go?
Go is primarily used for the development of web applications. The focus is on API and RPC services, dynamic HTML generation follows with a slight gap. Go is one of the leading programming languages in terms of container and cloud technology (e.g. Docker, Kubernetes, Terraform). A platform such as .NET is much more generally usable compared to Go, where other application areas such as UI clients, mobile apps, games etc. play an important role besides web programming.
Specialization is a double-edged sword for programming languages. On the one hand, it is a disadvantage because a company cannot consistently use one language for all application areas. On the other hand, it is an advantage because the language and the libraries and tools available for it can focus on a specific application area and are correspondingly good.
In my opinion, Go is very well suited for the development of microservices that offer RESTful or gRPC APIs. Go is a simply structured language that emphasizes consistency. Stability is especially important in Go, and APIs evolve more slowly in Go than in the fast moving UI world.
By generating native code instead of Intermediate Language, good start times and good runtime performance can be achieved. Go generates a compact executable file for a microservice, which – if you want it to – even runs without any external dependencies. The compiled gRPC client shown later in Listing 4 has without debug information only about 8MB and you don’t need to install any frameworks or other system requirements to run it. This compactness and performance are advantages when using microservices in cloud computing environments.
So there are good reasons to consider Go when developing gRPC Services.
ProtoBuf
The development of a gRPC API starts in Go as in all other languages with ProtoBuf files. ProtoBuf is a mechanism for language and platform independent serialization of data for its transmission and/or storage. The technology is therefore used similarly to JSON or XML, but it is much more optimized for performance through more compact, binary data structures. In addition to data structures, ProtoBuf also allows the definition of services (=APIs) that use the data structures for parameters and return values. These services become the gRPC API.
In this article I use the current version 3 of ProtoBuf (aka proto3). In practice you still find proto2 in existing applications, but new solutions should be based on version 3.
Installing the compiler
Enough with the theory, let’s look at an example. Before we can get started, we need to download and install the latest version of the ProtoBuf compiler (aka protoc) from GitHub. Don’t forget to add the compiler’s bin directory to the PATH environment variable so that protoc can be found on the command line. Then, we install the Go plugin for protoc with go get -u github.com/golang/protobuf/protoc-gen-go, which is responsible for generating go code. The Go plugin can be found in the directory $GOPATH/bin (or %GOPATH%\bin on Windows). Also make sure that this directory is referenced in the PATH environment variable.
You have already experimented with gRPC for .NET and wonder why the preliminary work at Go is so complicated? With .NET you had to do exactly the same thing, only Visual Studio did all the steps without you noticing. I use Visual Studio Code (VSCode for short) for Go development and therefore the above mentioned steps are necessary to set up the development environment. A tip for all those who use VSCode like me: There are some plugins for VSCode that add proto3 file support to the editor. Just search for ext:proto3 in the Extensions Manager and choose the one you want from the available VSCode extensions.
Simple proto3 file
Proto3 files always follow the same naming conventions regardless of the programming language used:
- File names follow the snake_casing_name scheme. The same applies to field names and enum values, whereby capital letters are used for the latter.
- CamelCasing is used for message names, enums and service names.
- We recommend that you use the keyword package to divide messages and services into namespaces so that identifiers are not used twice.
These naming conventions apply regardless of the programming language you use to program gRPC. The code generators will convert the ProtoBuf naming conventions into the appropriate naming conventions for the target language (for example conversion from camelCase to PascalCase).
Listing 1 shows a Hello World proto3 file that defines a service with one message for each parameter and return value. You compile it with protoc ./greet.proto –go_out=plugins=grpc:. The parameter value plugins=grpc tells the ProtoBuf Go plugin to generate gRPC compatible code. The result is a file greet.pb.go with generated Go code, which should not be modified, because it is generated every time the plugin is compiled.
Listing 1: greet.proto
syntax = "proto3";
// Language-specific option used go code generator.
option go_package = "greet";
package demo;
// The greeting service definition.
service Greeter {
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply);
}
// The request message containing the user's name.
message HelloRequest {
string name = 1;
}
// The response message containing the greetings.
message HelloReply {
string message = 1;
}
Worth mentioning in Listing 1 is the option go_package. The concept of ProtoBuf Packages cannot be directly assigned to Go Packages. Therefore, it is recommended to explicitly specify the go package for the generated code in proto files.
The Greeter.SayHello method shown in Listing 1 is a so-called Unary RPC Call. The client sends a single request and the server responds with a single response. gRPC can also handle client-side, server-side and bidirectional streaming. This will be discussed later in this article.
Generated Message Structures
The Go-code generated from the proto3 file is easily readable. Everything that starts with XXX_ can be ignored. These elements (e.g. fields in ProtoBuf messages) are necessary implementation details, but are not part of the public API.
The proto3 messages become Go structures (struct). The properties of the individual fields are mapped in Go as tags for the protobuf and json packages. Here is an example of the code generated from HelloRequest.Name: Name string `protobuf: “bytes,1,opt,name=name,proto3” json: “name,omitempty“`.
gRPC Service
A Go Interface is generated for each service of the proto file. In the case of the example from Listing 1, this is the GreeterServer interface. As a developer of the gRPC service, you have to implement the methods of the interface and do the actual work that the service is supposed to provide. Listing 2 shows a simple Hello World implementation of the gRPC service from Listing 1.
Listing 2: Implementation of the gRPC service
package main
import (
"context"
"fmt"
"github.com/Samples/BeyondREST/GoGrpcServer/greet"
)
type service struct {
}
func (s service) SayHello(ctx context.Context, req *greet.HelloRequest)
(*greet.HelloReply, error) {
// If you are new to Go, you can learn more about context.Context
// at https://blog.golang.org/context and https://golang.org/pkg/context/.
return &greet.HelloReply{
Message: fmt.Sprintf("Hello %s", req.Name),
}, nil
}
Listing 3 contains a simple variant of the code that can be used to start the gRPC server. In practice, this code will be more complex as you will add, for example, TLS encryption ( see e.g.), code for controlled shutdown of the gRPC server, or code for authentication (< a href=”https://github.com/grpc/grpc-go/blob/master/examples/features/authentication/server/main.go”> see e.g. ). The code in this article is intentionally kept simple so that we can concentrate on the core functions of gRPC.
Listing 3: Starting the gRPC server
package main
import (
"context"
"fmt"
"io"
"log"
"net"
"time"
"github.com/Samples/BeyondREST/GoGrpcServer/greet"
"google.golang.org/grpc"
)
[...] // Add code from Listing 2 (without import and package)
func main() {
listen, err := net.Listen("tcp", ":8080")
if err != nil {
panic(err)
}
// register service
server := grpc.NewServer()
greet.RegisterGreeterServer(server, service{})
greet.RegisterMathGuruServer(server, mathGuruService{})
// start gRPC server
log.Println("starting gRPC server...")
server.Serve(listen)
}
gRPC client
The client for our gRPC service was generated by protoc. We can use it without having to implement any interfaces. Listing 4 calls the gRPC service defined above. When calling the gRPC method SayHello, the context.Context must be considered. It has already been included in Listing 2. This type allows us to influence the behavior of the gRPC connection and to specify things like timeouts or to cancel the gRPC call.
Listing 4: gRPC Client
package main
import (
"context"
"log"
"github.com/Samples/BeyondREST/GoGrpcClient/greet"
"google.golang.org/grpc"
)
func main() {
conn, err := grpc.Dial("localhost:8080", grpc.WithInsecure())
if err != nil {
panic(err)
}
defer conn.Close()
client := greet.NewGreeterClient(conn)
response, err := client.SayHello(context.Background(),
&greet.HelloRequest{Name: "FooBar"})
if err != nil {
panic(err)
}
log.Println(response.Message)
}
Streaming
While the examples shown so far demonstrate the basic principle of gRPC in Go quite well, they only scratch the surface of what gRPC can do. In this article I would like to discuss an important, advanced feature of gRPC: Streaming. gRPC can do three variations of this:
- Server-side streaming: The server does not respond with a single response, but with a stream of responses.
- Client-side streaming: The client does not send a single request, but a stream of requests. The server responds with a response.
- Bidirectional streaming: The client initiates communication by calling the gRPC function and then sends a stream of requests. The server responds with a stream of responses. The two streams are independent of each other and can work interlocked. This means that the client does not have to have sent all the requests before the server begins to send responses.
Let’s have a look at an example. In this example, we return the numbers of a Fibonacci sequence from a gRPC server. To make things more interesting, let’s pretend that it takes longer to calculate the individual numbers. This allows us to use streaming. After all, the client can already process the first numbers, even though the server is still in the process of calculating the next results. In this example, the server is streaming responses to the client, so it is server-side streaming.
For space reasons, the rather simple source code for the iterator that calculates the Fibonacci sequence is not printed here. If you are interested, you can read it including unit tests on GitHub.
Listing 5 shows the extended proto3 file with the streaming service. Note in particular the keyword stream when defining the method GetFibonacci. It indicates that the method does not return a single response, but a stream of responses.
Listing 5: Proto-File with streaming
syntax = "proto3";
option go_package = "greet";
package demo;
[...] // Greeter Service shown in previous code samples
// Note streaming result
service MathGuru {
rpc GetFibonacci (FromTo) returns (stream NumericResult);
}
message FromTo {
int32 from = 1;
int32 to = 2;
}
message NumericResult {
int32 result = 1;
}
Listing 6 shows the implementation of the streaming method. As you can see, even with streaming, the code is not complicated. The degree of complexity of the code is not increased by gRPC or Go, it results from the application logic to be provided by the gRPC API.
Listing 6: Implementation with streaming
func (mgs mathGuruService) GetFibonacci(fromTo *greet.FromTo,
stream greet.MathGuru_GetFibonacciServer) error {
if fromTo.From > fromTo.To {
return status.Error(codes.InvalidArgument,
"FromTo.From must be <= FromTo.To")
}
fib := mathalgorithms.NewFibonacciIterator()
for fib.Next() {
if fib.Value() < fromTo.From {
continue
}
if fib.Value() <= fromTo.To {
stream.Send(&greet.NumericResult{
Result: int32(fib.Value()),
})
time.Sleep(250 * time.Millisecond)
continue
}
break
}
return nil
}
To demonstrate the platform independence of gRPC, in Listing 7 I show what a client for our streaming API in C# looks like. The gRPC client MathGuru.MathGuruClient was generated by Visual Studio 2019 from the same proto3 file we used for Go before (Listing 5). In the C# implementation it is noticeable that to consume the streaming API the new C# 8 function await foreach can be used. This makes the code very easy to read and write.
Listing 7: Consuming the Streaming API in C#
using Grpc.Core;
using Grpc.Net.Client;
using GrpcDemo;
using System;
using System.Threading.Tasks;
namespace GrpcClient
{
class Program
{
static async Task Main()
{
AppContext.SetSwitch(
"System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport",
true);
// Create channel.
// Represents long-lived connection to gRPC service.
// The port number(5001) must match the port of the gRPC server.
// Tip: In ASP.NET Core apps, use gRPC client factory
var channel = GrpcChannel.ForAddress("http://localhost:8080");
var mathClient = new MathGuru.MathGuruClient(channel);
await ServerStreaming(mathClient);
}
private static async Task ServerStreaming(
MathGuru.MathGuruClient mathClient)
{
// Note usage of new using statement (C# 8)
using var fibResult = mathClient.GetFibonacci(
new FromTo() { From = 10, To = 50 });
// Note usage of new streaming enumerable
await foreach (var fib in fibResult.ResponseStream.ReadAllAsync())
{
Console.WriteLine(fib);
}
}
}
}
STAY TUNED!
Learn more about API Conference
If you’re interested in what bidirectional streaming looks like, you can take a look at a more comprehensive version of the Fibonacci example discussed above on GitHub. There you can find a variant where the client sends requests for Fibonacci subsequences to the server via stream and the server streams the results back.
Standard Interface Definitions
If you try to implement a real API with gRPC, you will notice that you have to solve many problems that can be implemented with ProtoBuf, but where you would have to invent basic interface definitions yourself. Examples in this area are error codes and error messages, date values (ProtoBuf does not have a datatype built in for date values), monetary amounts with currencies, geocoordinates etc. Fortunately, Google provides the proto3 files on GitHub, which the company uses for its own cloud APIs. Since they are Open Source, they can be imported into your own projects. This way you save the time it would take to reinvent the wheel. In addition, consumers of the gRPC API find it easier to find their way around because they find structures they may already know from the Google APIs.
Conclusion
Go is a good programming language for implementing microservices. The code is consistent, stable over a long time and easy to maintain. The ecosystem of open source libraries for cloud native microservices is excellent, as Go is at home in this area. The resulting executables are performant, compact, have no dependencies and are therefore particularly convenient to use in container and serverless environments.
Microservices need a platform-independent interface to the outside world so that users are not forced into a certain technological direction. Here gRPC is a good choice. The code generators that generate code from proto3 files are available for practically all common programming languages and are of high quality. As hopefully you have seen from the examples shown above, developing gRPC code with Go is easy to learn. At runtime, gRPC offers comparatively good performance compared to RESTful Web APIs.
So there are many reasons to consider the combination of Go and gRPC especially in a cloud backend. gRPC is not a “REST killer” though. Outwardly to the web or mobile frontends one could use gRPC Web, but HTTP and JSON based web APIs (e.g. REST, GraphQL, JSON:API, OData) are often the better options in this area. In my opinion, today’s software development world no longer consists of monocultures in which one solution approach, one communication protocol, one programming language or one framework must be the solution for everything. The right combination of technologies and tools is what counts. In my view, Go and gRPC definitely have a place here.
